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Abstract 

The President of the University of Cincinnati 
charged the Academic Coordinating Committee (ACC), 
a universitywide governance body, to conduct a 
comprehensive review of degree programs by designing a 
process with multiple outcomes: building an academically 
stronger university; ensuring consistency with the 
institution's capacity and academic priorities; engaging 
individual units in a serious and focused examination of 
program mission, assessment practices, and plans for the 
future; and setting the stage for short- and long-term 
institutional planning. This article illustrates how the 
ACC envisioned, designed, and implemented the review 
process. It discusses our approach to developing and using 
a customized technology solution for program review. We 
believe that our experience and approach is transferable 
and can inform practices at other institutions. 
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Introduction 

The president of the University of Cincinnati 
charged the Academic Coordinating Committee 
(ACC), a universitywide governance body, to conduct 
a comprehensive review of all degree programs at the 
institution: associate, baccalaureate, graduate, and 
professional. She was especially concerned with what 
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was perceived to be a proliferation of degrees at all 
levels, and challenged the committee to design and 
implement a process for program review that would 
(a) address the number of degree programs in order 
to build an academically stronger university, more 
in keeping with peer and aspirant institutions; (b) 
ensure that degree programs were consistent with 
the institution's mission, capacity, and academic 
priorities; (c) engage individual units in a serious 
and focused examination of program mission, 
assessment practices, and plans for the future; 

(d) assist academic units in setting priorities for 
achieving excellence; and (e) set the stage for short- 
and long-term academic planning at the institution. 

University Profile 

The University of Cincinnati (UC) is a Public, 
Carnegie-classified Research University with 
Very High (RU/VH) research activity. It offers a 
comprehensive set of graduate and undergraduate 
programs, ranging from music and art to medicine 
and engineering, in 1 3 different colleges on its large 
urban campus. The University enrolls more than 
8,000 graduate students and 32,000 undergraduate 
students. It has a 15 to 1 student/faculty ratio and 
many ranked programs (http://www.uc.edu/about/ 
ucfactsheet.html). 

The ACC is a governance body that is chaired 
by the Senior Vice-Provost for Academic Planning. 
University administrators, associate deans, Faculty 
Senate representatives, and two student trustees 
serve on the committee. The administrators and 
associate deans are appointed by the Provost- 
Faculty Senate elects its representatives; and the 
student trustees serve by virtue of their position as 
student representatives to the UC Board ofTrustees. 

Rationale/Purpose 

Program review is a common practice among 
institutions of higher education. A preponderance 
of evidence suggests that program review has been 
primarily used as a means for monitoring the quality 


and effectiveness of educational programs (Barak 
& Breier, 1990; English, 1998; Thomas et al., 2000). 
Some have used the process to create a system of 
institutional accountability (Barak & Breier, 1 990; 
Pitter, 2007); others have used it to inform decisions 
about resource allocation and program closure 
(Michael, 1998). 

There is much debate in the literature on the 
uses and outcomes of program review. Whereas 
some articles focus on program review as a tool for 
program assessment and quality (Arns & Poland, 

1 980), others focus on the theoretical arguments 
for defining program quality (Bogue, 1 998; Green, 
1994) and benchmarking performance outcomes 
(Ferren & Aylesworth, 2001). Carter (2009) pointed 
out that the published literature on institutional 
assessment lacked a practice-based approach. We 
found a need for contemporary discussions on 
practical approaches that educational institutions 
develop to conduct program review, and on the 
corresponding challenges and outcomes that 
emerge from using such approaches. 

A Detailed Discussion of 
Our Review Process 

This article documents the process we 
developed at the University of Cincinnati to conduct 
universitywide program review, describes the 
challenges we faced, demonstrates how technology 
was used to conduct the review, and explains the 
decisions we made. It will clarify the rationale for 
those decisions and share the lessons we learned. 
We believe that the process we developed and the 
technology we designed could be adapted by other 
institutions and that our approach can be added to 
the inventory of ways to conduct institutionwide 
program review. 

The ACC conducted the review process in a 
staged manner. Each phase was time-driven with 
clear objectives and outcomes. These phases are 
shown in Figure 1 along with the questions we 
identified as important to address. These phases 
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Phase I: Establishing the framework for review 

• What are the objectives for program review? 

• How do we define a program for purposes of the review? 

• What specific programs will be reviewed? 

Phase II: Moving from principle to practice 

• How do we define program quality? 

• What specific program data points will help assess our definition of quality? 

• How will program data be collected? 

• From what institutional sources will program data be drawn? 

• How will collaboration between multiple data entities be established and streamlined? 

• How can the purpose and process of the review be aligned? 

Phase III: Orientation and public relations 

• What constituents within the university will be most involved or affected by the program 
review process? 

• How can faculty and administrators be brought on board with the review process? 

• How can angst of program closure be allayed? 

• What forums would provide maximum reach to create awareness about the purpose and 
process of the review? 

• How will program heads be oriented on the data entry process? 

Phase IV: Data collection 

• What administrative structures will be used to ensure quality and timely input of program 
data? 

• What are reasonable deadlines for program data entry? 

• Who will be responsible for user support and troubleshooting user questions? 

Phase V: Review of program data 

• Who will review program data? 

• What guidelines will be used to review programs? 

• What are reasonable deadlines for reviewers? 

• Should programs have an opportunity to respond to the initial outcomes of the review? 

Phase VI: Implementation and resource alignment 

• How will review outcomes be disseminated to college deans? 

• What concrete steps should follow program review? 

• Should the outcomes of program review be made public? 


Figure 1: Questions we asked while planning and conducting program review. 
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were more layered than might appear and some 
more complex than others. For example, the ACC 
spent several months working through Phases l-lll 
to articulate the guidelines for these aspects of the 
review, whereas, the actual data collection took 
much less time. Figure 2 documents the timeline for 
the phases in our review process. 


detail that is consistent with review objectives. 
Second, it should reflect how programs are 
structured and how they operate across different 
colleges in a university. Third, the definition must 
consider how program data like student enrollment, 
faculty FTE, or retention rates are consolidated at 
the institutional level. If program review involves 


Program Review Timeline 



Duration 


Figure 2. Timeline of the review process. 


Phase I - Establishing the Framework for 
Program Review 

Our first task consisted of clarifying review 
objectives, envisioning a review process, and 
solidifying a strategic approach to program review. 
These first steps were critical for the review process 
to be successful and for it to connect with and 
influence other institutional priorities. 

Defining a "program" for purposes of the review 
proved challenging. There are myriad ways to define 
a program including degrees, majors, minors, tracks, 
concentrations, or certificates. More generally, we 
found that a program could encompass an entire 
department housing several such academic entities. 

Practically, the definition determines the unit of 
analysis for the review and, thus, the list of programs 
to be reviewed. We found that this aspect of the 
definition has several implications for the process. 
First, the unit of analysis must provide a level of 


financial data, the extent to which these data can be 
broken down further determines the definition of a 
program. 

To address these issues, we conducted an 
audit of all degree codes in partnership with 
College Deans, the Registrar, and our Institutional 
Research Office. The goal of the audit was to identify 
duplicate degree programs within and across 
academic units and to derive a list of programs 
for review. Through the audit, several inactive 
program codes were identified and either closed 
or consolidated. By consolidating or eliminating 
inactive programs, the e-review process also helped 
us streamline our submission of the Academic 
Program (AP) file to the Ohio Board of Regents, 
Higher Education Information System. 

Based on the above analysis, we defined a 
program as "degree granting" despite the many 
ways in which "programs" are coded in the student 
information system. Using this definition, we 
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determined that the review process would involve 
293 degree-granting programs: 159 undergraduate 
programs and 1 34 graduate programs. The 
program audit allowed us to formalize our method 
for reporting program information to external 
agencies. By defining the programs to be reviewed, 
the review process allowed us to consistently and 
accurately report program information on the 
Higher Learning Commission's Annual Institutional 
Data Update. 

Our approach was logical and compatible 
with the university structure but created its own 
challenges. For instance, from an institutional 
perspective, this definition excluded certificate 
programs and minors. Since an objective for 
program review was to streamline program 
offerings, the departmental resources expended 
to operate and sustain certificate programs did not 
show up on the radar of program review. 

Phase II - Moving from Principle to Practice 

The program list that emerged through the 
degree audit provided the structure for Phase II, 
which involved the following: defining program 
quality; identifying criteria to assess a program; 
establishing a means to collect, review, and report 
program data; and developing workflows and 
timelines. 

Developing criteria to assess a program. The 

wide disciplinary expertise and representation of 
faculty and administrators on the ACC facilitated 
discussion and consensus-building on the broad 
themes and criteria to be used to assess programs. 
Scott (1 980), Dill (1 992), and Michael (1 998) held 
that program quality is difficult to define, as it 
is an incomprehensible concept. Bogue (1 998) 
questioned, "Is the quality of an educational 
program... to be found in its reputation or in its 
results, in rigor of process or in proof of outcome?" 
(p. 7). We found it helpful to take a comprehensive 
approach to program quality. 

As advocated by Garvin (1 988), the ACC selected 
a multifaceted definition of program quality that 


brought together several theories. The first revolved 
around the articulation of a program's mission and 
objectives and whether the program was meeting 
its stated mission within the context of college and 
university goals (Green, 1994). The second involved 
identifying internal and external quality assurance 
mechanisms and establishing how such assessment 
mechanisms are used for program improvement 
and strategic planning (Deming, 1 986). The third 
focused on performance or outcomes as it related 
to students, faculty, and the program as a whole 
(Guaspari, 1985). The fourth focused on a program's 
vision and plans for the future and how those plans 
meshed with the mission of the program. 

Establishing criteria to assess program 
quality. Sub-committees of the ACC researched 
each of the above quality themes and outlined the 
nature and parameters of individual criterion that 
could help assess each quality theme. The ACC, as a 
whole, refined each criterion and worked together 
to articulate its understanding of each criterion 
so that it would be accurately understood and 
interpreted during the data entry and review stages. 
These criteria are discussed in detail later in the 
article. 

Given the size of the institution and the diversity 
of program offerings, we found it important to 
ensure that the review criteria covered a wide 
array of program data so that each college had a 
fair chance to showcase its programs. At the same 
time, it was important for program data to provide 
reviewers with sufficient evidence to distinguish 
among strong and weak programs and reveal 
areas for improvement. Consistent with counsel 
from Ferren and Aylesworth (2001 ), we sought a 
combination of both quantitative and qualitative 
data, including the areas that follow. 

Mission and objectives. To assess quality in 
this category, each program articulated its mission 
and objectives and identified distinctive features 
that the program employed to achieve its mission 
and objectives. The ACC identified a list of program 
features (see Figure 3) like service learning, study 
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Program Features 


□ Internship 

□ Clinical Placement 

□ Co-op international 

□ Co-op domestic 

□ Performance 

□ Exhibitions / Creative Work 
D Undergraduate Research 

□ Laboratory / Studio 

□ Other Experiential Learning 


□ Domestic cultural competence 

□ Global cultural competence 

□ Study Abroad 

□ Service Learning domestic 


□ Service Learning international 

□ Problem- or case-based instruction 

□ Instructional Technology _J 


D Distance Learning 
D Library Research Component 

D Significant student writing (at least a four page paper) 

□ Student presentations (in-class) 

□ Student presentations (out of class including professional meetings) 

□ Corporate collaborative 

□ Interdisciplinary connections or perspective 

□ Students are members of professional organization 

□ Other 


Figure 3. Program features prompts in e-review. 


abroad, internships, or the use of instructional 
technology, and units could use the list or add to 
it features that were unique to the program. Each 
program was required to highlight how it achieved 
its mission and how the mission aligned with the 
institution's academic plan and position as an urban 
research university. 

Internal and external program assessment. 

Program quality assurance has been traditionally 
ensured through the use of both external and 
internal assessment measures like accreditation 
and peer review (Arns & Poland, 1980). The ACC 
decided to use the results of internal and external 
assessments to evaluate programs. Programs 
were requested to provide documentation on the 
outcomes and recommendations of accrediting 
agencies together with evidence of the program's 
response to those recommendations. Programs 
presented data on accreditation cycles so that 
reviewers would be aware of whether a program 
had recently undergone an accreditation visit 
or whether a visit was forthcoming. Graduate 
programs provided the most recent "closure" letter 
from the institution's graduate review process; 
these letters summarized the findings of the review 


process and outlined steps for improvement to be 
addressed prior to a future review. 

Since all programs do not have specialized 
accreditation, the ACC requested data on 
internal assessment mechanisms. Similarly, those 
programs that had undergone review by internal 
or external peers attached the outcomes and 
recommendations of such reviews and responses by 
the program. 

The ACC developed a comprehensive list of 
assessment practices used within the university to 
maintain program quality (see Figure 4). Programs 
could use the list as a base to indicate which 
practices formed part of their assessment methods 
or add assessment practices that were not listed 
or were unique to a program. In addition, the ACC 
sought evidence-based explanations on how 
assessment practices informed current operations 
and future plans for the program. 

Performance outcomes. The use of program 
performance results for program review is not 
without controversy. While some like Bogue (1 998) 
have held that performance assessments and 
evidence are required to make a fair judgment of 
program quality, others like Arns and Poland (1 980) 
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Assessment Practices 

D Standardized test D General Education assessment 

□ Skill proficiency testing 


□ Portfolio assessment 



D e-portfolio assessment 


□ Internal Review 

D Pre- / post testing 


□ External Review 

□ Focus 

groups 


□ Other 

D Professional certification 



By Whom: 

Pass Ra tes: Na tional A vg 


□ 


By Whom: 

Pass Rates: National Avg: 


□ 



Briefly indicate how the assessment data that you collect is used to make program improvements: 


Figure 4. Assessment practices prompts in e-review. 


questioned whether outcomes like a program's 
contribution to knowledge or skill creation, 
creativity, or to a society or community can be 
quantified. However, they asserted it is possible 
to find "quantifiable surrogates" (1980, p. 271) for 
performance outcomes. 

To assess performance outcomes, the ACC 
sought a combination of both quantitative and 
qualitative data in three categories — student, 
faculty, and institutional outcomes. With regard to 
student outcomes, quantitative data like enrollment 
history, graduation rates, and retention rates for 
the last five years were extracted from records in 
the office of institutional research (see Figure 5). 
Program heads supplied information about the 
demographics of students to which the program 
caters and employment rates of students upon 
graduation or any time period after graduation 
(depending on how the college collected data). 
Undergraduate programs were asked to indicate 
how the program prepared students for graduate 
school. 

For faculty outcomes, the ACC requested data 
on the ability of a program to attract and retain the 
best faculty in the market. Quantitative data like 


faculty full-time equivalency was extracted from 
institutional research records. Programs described 
the volume of faculty research and grants and the 
involvement of students in research activities. 

The third performance category related to 
the institutional competitiveness of the program. 
Programs provided evidence-based data of rankings 
by the U.S. News & World Report and ratings by other 
organizations. Where applicable, programs provided 
the results of student performance on professional 
and licensure tests and comparisons with national 
averages on such tests. 

We decided to use performance results to 
frame the discussion on program performance 
rather than to define it. For instance, we found 
that our program offerings are built around 
disciplines rather than function. Some programs 
transcend disciplinary boundaries and serve 
students outside their discipline, department, or 
college structure. As a result, such programs are 
likely to show low enrollment numbers in the 
major but have high faculty utilization because of 
the large number of students served. Therefore, 
we also sought qualitative data that addressed 
service, collaboration, and partnerships that 
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J IR Data 


Fall 2003 

Fall 2004 

Fall 2005 

Fall 2006 

Fall 2007 

(Enrollment Headcount 


1 1 

1 

(instructional FTE 


1 

1 

[First Year Retention Rates 






| [Graduation Rates 







□ Comments: 


Figure 5. Institutional research data interface in e-review. 


programs had built within the university, with other 
universities, and in the community. Our discussion 
on developing review criteria took into account 
programs that serve to diversify the university's 
portfolio. 

Future plans and vision. Each program was 
asked to list at least three strategic action steps 
that would propel the program toward excellence. 
Programs were required to provide explanations 
on the linkages between future plans and program 
mission and vision. 

Establishing the means to collect program 
data. After evaluating the traditional means used 
to collect program data, we determined that paper 
or email options would be cumbersome, require 
excessive print and paper resources, and make 
the process of consolidating and reporting review 
outcomes tedious. More recently, some universities 
have used electronic options like e-portfolios to 
collect program data. While e-portfolios work well 
for the collection of program data and may be good 
repositories of program data, it would be difficult to 
query, track, and extract meaningful reports from 
the data. This limits the potential of the enormous 
effort and resources that universities expend in 
conducting program review. Buying a commercial 
product was an option but we felt it would hamper 
the flexibility of our approach and be restrictive 
on the customization of information for reporting 
purposes. In addition, commercial products would 


place restrictions on the nature, type, and source 
of institutional data that we could bring into the 
program review process. 

We determined that a customized, web- 
enabled, database management system tailored 
for program review would best fit our needs to 
streamline the data collection and review process 
across the university and allow for intelligent 
querying, tracking, and reporting. The system was 
named e-review. The ACC wanted e-review to be 
user-friendly, secure, and compatible with existing 
university systems. A key aspect in developing 
customized software is selecting a database system 
and development platform. We wanted a robust, 
relational database management system (RDBMS) 
and a development platform that was compatible 
with other university systems with which e-review 
had to interface. We needed to easily find 
personnel with programming expertise in the 
platform we selected. 

Based on these considerations, e-review was 
designed and developed as a relational database 
management system with a web-enabled front end. 
We used ColdFusion 8® to develop the application 
layer, or front-end, and Microsoft SQL 2008® for the 
database back-end. Development was done within 
the Fusebox 5.0® framework. The Provost Office 
allocated $25,000 to hire a program developer and 
provided the resources to hire a graduate assistant 
for two years (including summers) at $20,000 per 
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year. Both individuals had expertise in program 
design and technology and worked closely with 
the ACC to design, develop and implement the 
technology solution called e-review. 

e-review comprised three front-end modules — a 
data entry module, a reviewer module, and a 
reports module. Because our review process was 
internally driven and reviewers were part of the 
university structure, system security was very 
important. To address system security, we made 
e-review available only on the University's intranet; 
built access to e-review around the established 
university network authentication system; and 
protected reviewer identity. We created a hierarchy 
of three user groups — administrators, reviewers, 
and program data entry users — that received access 
to specific system functions and data types. 

We built interfaces between e-review and other 
existing university systems to upload and hand off 
program data during and after the review. This was 
accomplished through collaboration among the 
Provost Office, Registrar, and Institutional Research. 
With an eye to the future, it was also vital to keep 
the database structure scalable so that e-review 
could be expanded with future enhancements. 

Establishing workflows and timelines. As 
a process, the ACC used existing administrative 
structures and hierarchies for communication 
and reporting channels. When conducting a 
project of this scale, we found it helpful to plan for 
contingencies and delays and to be flexible with 
deadlines as some colleges or programs may be 
involved with other parallel and time-sensitive 
activities. An analysis of Figure 2 reveals that we 
spent a considerable amount of time on planning, 
development, and testing. Our experience revealed 
that good planning is crucial to executing a 
smooth and effective process and is a guarantee to 
successful program review. 

Phase III - Orientation and Public Relations 

In Phase III, we presented the review purpose 
and process to the university community. Arns 


and Poland (1 980) pointed out that perceptions 
and motivation play a large role in the success of 
program review. One of the challenging public 
relations hurdles we encountered related to a 
perceived link between program review and 
program closure. 

Through presentations to administrator and 
faculty groups on campus, we sought to gain 
cooperation from those in leadership positions at 
the colleges (Bogue, 1 998), gauge initial reactions 
to e-review as a system, and involve faculty by 
requesting feedback on the template (Kleniewski, 
2003). We used these efforts to allay fears of 
program closure by having senior administrators 
clarify that the program review process was not 
about program closure. Our public relations efforts 
made positive connections between the review 
purpose and process. We identified opportunities 
for linkages of program review efforts with other 
college and university priorities, including the 
preparation of the HLC/NCA Self Study and on-site 
visit, collegiate restructuring, and the conversion 
from a quarter to semester calendar. 

Phase IV - Data Collection 

We launched the data collection phase in two 
stages, beginning with graduate and professional 
degree programs followed by undergraduate 
programs. We incorporated several user-friendly 
features into e-review such as using a combination 
of text fields and check boxes to aid fast data entry. 
Wherever possible, e-review provided for the upload 
of evidence or documents, which limited the need 
for users to enter copious amounts of data that 
already existed electronically. Users could cut and 
paste information from other Microsoft Office® 
products into e-review. Data filters helped users 
easily query and sort programs. 

The interfaces built between e-review and 
existing data sources in the university allowed 
for institutional data to be pre-populated into 
each program template. This minimized the need 
for user input and ensured the accuracy of data. 
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Because e-review was available on the university 
intranet and allowed for off- and on-campus user 
access, a help system provided detailed instructions 
for users to login and use the program. This 
established a common user experience across the 
university. 

At any stage during data entry, users could click 
a button to save data incrementally or generate 
a printable, formatted .pdf document with the 
latest data entered in the system. Program data 
were deemed ready for review after college deans 
indicated it was internally consistent and ready for 
evaluation. This was accomplished with the click 
of a button. Faculty uncomfortable with using the 
web-based technology presented a challenge. 

Thus, the technology team became a helpdesk and 
responded promptly to user queries or requests 
for support to create a positive user experience. A 
screen shot of the data entry screen can be found at 
http://uc.edu/provost/ereview 

Phase V - Review of Program Data 

We conducted the actual review in two stages. 
We decided that members of the ACC would 
review program data as they represented a good 
blend of disciplinary expertise, diversity, campus 
representation, and they best understood the 
intended outcomes of the review process. In 
addition, we anticipated that an institutional profile 


would become clearer if a collaborative body was 
involved in conducting the review. 

In order to establish broad reliability of 
outcomes among reviewers, each ACC member 
individually reviewed three sample programs and 
made (a) a determination as to the completeness 
of the response relative to the template; and (b) a 
judgment on the information submitted as to where 
a program was relative to quality. The committee 
then met as a whole and discussed the ratings of 
reviewers (satisfactory, unsatisfactory, or needs 
more information) and debated justification for 
their assessment of the programs. This helped us 
recognize the range of reviewer responses and 
work collaboratively with the entire committee to 
develop consensus among reviewers on evaluating 
program data. 

In the first stage all 293 programs were 
reviewed. ACC members divided into teams taking 
into account disciplinary expertise and conflict of 
interest. First, each member on a team reviewed and 
rated a program individually and then collaborated 
with members on their team before deciding on 
the rating of the program. The reviewer interface 
of e-review was designed for multiple reviewers 
to view the same program data set (see Figures 
6 and 7). Reviewers could review program data 
individually and then collaboratively assess a 
program. 



Figure 6. Part of the reviewer summary screen. 
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Review Committee 


Review Conclusion 


Comment Log: 

First review: Review Indicator: 

The information provided in the e-Review and the Roadmap document (enclosed) do not First Review: Unsatisfactory 
seem to be well aligned. The majority of the responses provided are unclear or less than 

adequate. Does the doctoral program have an account of student learning outcomes and Second Review: Satisfactory 
strategic plan document. 


Second review: 

Thank you for the extensive additional response material. The assessment matenals are very 
comprehensive. It remains very difficult to distinguish the masters program from the doctoral 
program. 


This program was resubmitted for 2008 review on 12/22/2008 
Print (PDF) | 


Figure 7. Sample reviewer comments in e-review. 


In particular, ACC reviewers expected each 
program to provide a concise and honest appraisal 
of its strengths and weaknesses, and its plans to 
improve. The degree to which this was present 
in each unit's submission affected how the ACC 
rated the program. For instance, it was important 
to not see the program mission as an isolated 
unit, but to analyze it within the context of its 
relationship to student and faculty outcomes, and 
to evaluate how future plans for the program were 
in keeping with the spirit of the program mission. 
ACC reviewers looked for an action plan linked to 
a program's overarching mission that would help 
increase stature or improve competitiveness. The 
use of both quantitative (e.g., retention and time- 
to-degree percentages) and qualitative date (e.g., 
rankings, external reviews, accreditation reports) 
provided reviewers with information that could be 
used to understand a program's current status, its 
health, and its potential for moving forward and 
achieving excellence. 

Each review team presented its findings to 
the ACC as a whole, which then collaboratively 
discussed and deliberated on problematic program 
data. This helped reviewers examine the health 
of individual programs and at the same time 


acquire a holistic understanding of the university's 
program offerings in order to evaluate the congruity 
of program offerings within a department and 
across colleges at the university. In the first stage, 
programs were assessed into three categories based 
on these considerations: 

• Satisfactory - Program ranges from good 
to high in quality. Program has potential to 
enhance standing of university. Program is 
doing reasonably well with viable plans for 
moving forward. Program demonstrates 
through its action plan a strategic agenda 
for achieving excellence. 

• Needs more information - Data provided by 
the department heads in e-review are not 
complete. Program needs to furnish missing 
data. Program is new or developing. 
Appears to duplicate other existing 
programs. 

• Unsatisfactory - Program has significant 
problems or critical issues in terms of 
structure, mission, or internal or external 
assessment outcomes. Program has critical 
issues moving forward and there is no clear 
plan for how it will achieve excellence. 
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In an effort to be fair in its assessment, the ACC 
requested all programs marked with unsatisfactory 
or needs more information to revisit the program 
data template and respond to reviewer comments/ 
concerns or provide additional evidence for 
clarification. The ACC identified model programs in 
each college that had provided evidence-based and 
complete responses in the template. Programs that 
were identified as unsatisfactory or needing more 
information were requested to review submissions 
from model programs before undertaking data 
entry for the second time. 

Interestingly, during this stage, colleges 
themselves recommended programs for closure. 
Their decision was based on factors like low 
enrollment, lack of potential for achieving 
excellence, and low market demand. Any resources 
that were saved were reallocated within the college 
to stronger and more competitive programs; faculty 
were reassigned within the college. It must be 
noted that the closures were not recommended by 


the ACC as the reviewing body. The process of self 
evaluation by the program units led to the closures. 

Therefore, in the second round of review 
we used the following review categorizations: 
satisfactory, unsatisfactory, and college recommends 
for closure. Each college received a report 
generated in e-rew'ewwith a summary of the review 
outcomes. In addition, users could log in to e-review 
and view a detailed analysis of the program by 
reviewers. The universitywide outcomes of program 
review are depicted in Figure 8. 

Discussion 

Customized Approach 

We believe that one of our best decisions was to 
embrace a customized approach to program review. 
The processes we developed to collect, review, 
and make changes to the portfolio of programs 
were grounded in the university's unique culture, 


Program Review Final Outcomes - Total 293 

15 



■ Satisfactory 

■ Unsatisfactory 

College recommends for 
Closure/ Consolidation 


258 


Figure 8. Summary of final outcomes of program review. 
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local operations, and organizational structures. The 
ACC's use of existing administrative and reporting 
structures within the university as communication 
channels ensured that the process was taken 
seriously and provided for accountability at various 
levels. 

Technology-driven Approach 

Using customized technology provided 
enormous flexibility and yielded multiple dividends. 
Because of its usability and convenience, e-review 
leveraged the data entry and review processes. It 
facilitated a smooth flow of information among 
the various partners, reduced the need for copious 
paperwork, and eliminated printing costs. Tracking 
and reporting features of e-review allowed the ACC 
to efficiently monitor and oversee the entire review 
process. Institutionally, e-review has become a one- 
stop, electronic snapshot of the university's degree 
offerings. We have found that well-coded program 
data are essential for intelligent reporting and they 
are informing our short- and long-term institutional 
planning. 

The ACC decided to henceforth conduct 
universitywide program review in 5-6-year cycles. 
Because e-review is scalable, we can easily enhance 
or expand the system in the future with new 
features. It will serve as the foundation for future 
program review cycles at the university, and for 
the development and enhancement of program- 
specific assessment plans. 

Fair and Comprehensive Approach 

Our experience reaffirmed Barak and 
Breier's (1 990) findings that fairness, objectivity, 
comprehensiveness, and credibility are key 
principles for successful program review. For us, a 
comprehensive approach meant building criteria 
that showcased the university's diverse portfolio of 
programs. It meant the reviewers would see each 
program holistically and evaluate the congruity of 
program offerings within a department, college, 


and the university. The diversity of representation 
and disciplinary expertise represented on the 
reviewing body facilitated the development of 
thoughtful criteria for evaluation. At the same time, 
it created opportunities for healthy differences 
in opinion. Our consensus-based approach to 
evaluating programs along with the criteria-based 
nature of the program data template provided for 
an objective framework for the review process. 

Collaborative Approach 

Our approach brought together faculty and 
administrators in deliberate and meaningful 
collaboration at various stages of the review process. 
They collectively developed the criteria used to 
assess program quality. We observed positive levels 
of collaboration during the data entry and review 
phases. The handoff of program data from data 
entry users to reviewers and vice versa was achieved 
seamlessly with the click of a button. Through 
the outreach and public relations efforts of ACC 
members, faculty came to appreciate the importance 
of the review process and seized the opportunity to 
showcase their programs at the university level. All 
these factors helped alleviate what Arns and Poland 
(1980) described as "snags" (p. 274) that can occur 
in conducting program review when faculty and 
administrators come together clumsily. 

Immediate, Windfall, and Institutional 
Outcomes 

We found it helpful to classify outcomes 
of program review into three categories: 
immediate, windfall, and institutional. We used 
the results of program review to make immediate 
recommendations to the Provost about program 
consolidations, closures, and realignments. Thanks 
to the design of the review itself, academic units 
were asked to align academic mission and strategic 
priorities resulting in an accurate and timely 
catalogue of degree program offerings. The data 
we collected were also used to realign budgetary 
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resources within a performance-based budgeting 
model implemented since the 2010 fiscal year. 

The versatility of program data that are 
stored electronically means that data can be 
easily tracked or spun into customized reports 
long after the review process is completed. As a 
result, we continue to reap windfall benefits from 
program review. Consider that when the Office for 
Assessment and Student Learning needed data on 
programs across the university that offered service 
learning or programs that were professionally 
certified, those data were made available at the click 
of a button as a report. The review outcomes are 
being used to inform decisions on the development 
of the university's library collection. 

We found it valuable to align the purpose of 
program review with other institutional endeavors 
(Barak & Mets, 1 995). The results of the review 
provided the framework for the three large 
institutional initiatives mentioned previously: the 
self study and on-site visit of the HLC/NCA, the 
ACC's involvement in a universitywide college 
restructuring endeavor to combine and synergize 
program offerings for greater efficiency, and the 
process of migrating from the quarter to semester 
academic calendar. This alignment served to place 
the review process in perspective, generated 
motivation among faculty, and made the data 
collection and review process meaningful beyond 
its immediate scope. 

Program Closure/Public Relations 

Whether intended or not, program review 
creates angst of program closure. Programs 
generally respond to such fears by posturing 
defensively or withholding important data. We 
found it helpful to make public relations attempts 
to squelch such fears early in the process. The 
use of generic review outcome classifications 
like satisfactory, unsatisfactory, and needs more 
information in the first stage of the review allayed 
fears and clarified that the review process was not 
about program closure. We found it a good practice 


to give programs an opportunity to respond to 
reviewer comments or provide additional evidence 
to clarify concerns. Decisions on the health 
of a program were made only after programs 
had responded to initial concerns raised by the 
reviewers. This iterative process is powerful in itself, 
and the programs that were closed after the review 
were recommended for closure by the colleges 
themselves. 

Our approach to program review broadly 
involved evaluating academic strength, internal and 
external assessment mechanisms, and performance 
outcomes as indicators of program strength. 
Conspicuously missing were cost and revenue 
data and criteria. It is difficult to access cost and 
revenue data at the program level in our financial 
system because university budgets operate on a 
departmental basis, and the blurred lines between 
a program's structure and function make cost- 
based criteria difficult to assess. Certainly, this was a 
limitation of our process, but accessing these data 
remains a goal with the implementation of our 
performance-based budgeting model and will be 
added as an interface to e-review in future program 
review cycles. 

Conclusion 

As the demands for accountability in higher 
education increase from multiple stakeholders, we 
believe that our approach to program review, and 
the system we developed, aids in the assessment 
of program quality and facilitates long- and short- 
term institutional planning. The process and 
means that we developed can be adopted by other 
institutions investigating ways to develop practical 
and useful approaches to program review at the 
level of the university. Our process was highly 
respected at UC since it was viewed as inclusive 
and fair, and the results were used extensively for 
regional accreditation by NCA/HLC, for semester 
conversion, and for performance-based budgeting. 
It has served as a vehicle for internal planning and 
external reporting. 
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